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Abstract 

The  purpose  of  this  paper  is  to  analyze  the  probabilistic  cost  model  currently  in  use  by 
NAVSEA  05C  to  predict  cost  uncertainty  in  naval  vessel  construction  and  to  develop  a  better 
method  of  predicting  the  ultimate  cost  risk.  The  data  used  to  develop  the  improved  approach  is 
collected  from  analysis  of  the  CG(X)  class  ship  by  NAVSEA  05C.  The  NAVSEA  05C  cost  risk 
factors  are  reviewed  and  analyzed  to  determine  if  different  factors  are  better  cost  predictors. 
The  impact  of  data  elicitation,  the  Money  Allocated  Is  Money  Spent  (MAIMS)  principle,  and 
correlation  effects  are  incorporated  into  the  research  and  analysis  of  this  paper.  Data  quality  is 
directly  affected  by  data  elicitation  methods  and  influences  the  choice  of  probability  distribution 
used  to  give  the  best  predictor  of  cost  risk.  MAIMS  and  correlation  effects  are  shown  to  make  a 
significant  impact  to  the  overall  cost  model.  Program  managers  and  analysts  can  readily 
implement  the  enhanced  models  using  commercial  Excel®  add-ins,  such  as  Crystal  Ball®  or 
@Risk  and  integrate  them  into  their  current  cost  risk  analysis  and  management  practices  to 
better  mitigate  risk  and  control  project  cost. 

Executive  Summary 

In  order  to  generate  the  funds  to  implement  the  30-year  plan  of  future  ships  and 
capabilities,  the  Navy  must  explore  different  options  for  cost  savings.  Fundamental  to  the 
success  of  complex  projects,  such  as  naval  vessel  construction,  is  the  ability  to  control, 
manage,  and  communicate  the  status  of  the  risk  reduction  effort  throughout  the  development 
and  production  cycles  (Kujawski  &  Angelis,  2007).  It  is  recognized  that  the  Navy  and  the 
shipbuilding  industry  need  to  change  their  technical  and  business  shipbuilding  strategies  in 
order  to  achieve  the  goal  of  a  future  fleet  that  balances  both  capability  and  affordability.  Cost 
risk  assessment  and  analysis  is  one  tool  that  can  be  utilized  to  help  recapitalize  costs  used  in 
the  ship  acquisition  and  building  process. 
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This  paper  analyzes  the  probabilistic  cost  model  currently  in  use  by  Naval  Sea  Systems 
Command  Cost  Engineering  and  Industrial  Analysis  Division  (NAVSEA  05C)  to  predict  cost 
uncertainty  in  naval  vessel  construction  and  to  develop  a  method  that  better  predicts  the 
ultimate  cost  risk.  The  NAVSEA  05C’s  cost  analysis  model  for  the  proposed  new  cruiser, 

CG(X),  encompasses  all  aspects  of  cost  for  the  entire  fleet,  including  inflation  and  profit.  The 
data  used  in  the  NAVSEA  model  were  acquired  from  subject  matter  expert  (SME)  inquiry  using 
three-point  estimates  of  high,  most-likely,  and  low  values.  The  Navy  is  placing  great  emphasis 
on  producing  the  best  product  for  each  dollar  spent.  In  order  to  ensure  the  continued 
acquisition  of  CG(X),  it  is  important  that  realistic  cost  risk  analysis  be  performed  so  that  program 
managers  can  make  informed  decisions. 

The  cost  model  elements  investigated  in  this  paper  include  data  elicitation  methods, 
probability  distribution  function  (PDF)  choice,  correlation  effects,  and  Money  Allocated  is  Money 
Spent  (MAIMS)  principle  effects.  The  most  significant  impact  is  seen  with  MAIMS  and  data 
elicitation  effects.  PDF  choice  and  correlation  effects  have  lesser  impact  upon  the  cost  model. 

Methods  of  data  elicitation  are  explored  and  the  use  of  a  direct  fractile  assessment 
(DFA)  is  recommended  for  future  use  (Kujawski,  Alvaro  &  Edwards,  2004),  although  the 
research  in  this  paper  does  not  involve  data  acquisition.  To  simulate  the  use  of  a  DFA 
methodology,  three-parameter  Weibull  distributions  are  employed  to  account  for  uncertainty 
associated  with  SME  estimation  of  data.  A  Weibull  (10%,  50%,  90%)  distribution  is  used  to 
simulate  a  more  optimistic  view  of  the  uncertainty  of  data,  while  a  Weibull  (20%,  50%,  80%) 
distribution  models  a  more  pessimistic  view. 

The  methodology  in  choosing  different  probability  distribution  functions  and  their 
applicability  to  the  model  is  evaluated.  Specifically,  triangular,  lognormal,  and  two  variations  of 
the  three-parameter  Weibull  distribution  are  considered.  Once  enhanced  models  are 
established,  program  managers  can  implement  them  into  their  current  cost  risk  analysis  practice 
to  mitigate  risk  and  control  project  cost. 

Two  types  of  correlation  effects  are  considered  and  modeled  in  this  paper.  The  first  is 
the  correlation  between  the  components  of  the  radar  suite,  and  the  second  is  the  correlation 
between  all  the  components  of  the  electronics  suite.  The  radar  suite  is  one  of  the  systems  that 
make  up  the  electronics  suite.  The  results  suggest  that  the  correlation  effects  are  important  for 
probability  values  midway  between  the  mean  and  the  extremes,  but  there  is  little  difference  for 
correlation  coefficients  beyond  0.5.  Further  investigations  are  recommended  to  quantify 
correlation  effects. 

MAIMS  modified  probability  distributions  are  used  to  show  the  significance  of  budget 
allocation  levels  (Kujawski,  Alvaro  &  Edwards,  2004).  These  distributions  reflect  an  empirically 
observed  effect,  namely,  that  once  a  budget  is  allocated,  the  project  cost  will  most  likely  be  at 
least  equal  to  the  amount  allocated.  As  the  MAIMS  modification  value  increases,  the  overall 
distribution  cost  rises  with  increasing  probability. 

Credibility  and  realism  are  two  key  cost  risk  assessment  criteria.  The  use  of  improved 
methods,  such  as  those  investigated  in  this  paper,  are  especially  significant  for  today’s  Navy 
during  a  time  of  budget  hardship.  If  the  Navy’s  plans  for  a  31 3-ship  fleet  are  to  become  a 
reality,  the  incorporation  of  cost  risk  analysis  into  acquisition  and  shipbuilding  management  is 
imperative.  Reliable  cost  assessments  can  help  deliver  projects  on  time,  at  a  lower  cost,  with  a 
higher  probability  of  success.  Effective  training  of  personnel  involved  in  cost  assessment  and 
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continued  efforts  to  improve  existing  cost  models  will  help  improve  the  Navy’s  current  cost 
estimating  process. 

I.  Introduction 

Admiral  Gary  Roughead  stated  in  the  Chief  of  Naval  Operation’s  (CNO)  Guidance  for 
2007-2008  that: 

We  manage  risk.  We  will  identify,  analyze,  mitigate  and  then  accept  risk, 
appreciating  that  we  must  always  consider  the  risks  in  aggregate  across  the 
entire  force.  Zero  risk  is  not  achievable  nor  affordable.  We  must  manage  risk 
and  move  forward  to  accomplish  the  mission  while  safeguarding  our  people  and 
infrastructure.  (Roughhead,  2007) 

Vice  Admiral  K.  M.  McCoy  took  this  a  step  further  in  2008,  in  a  statement  made  on 
assuming  the  position  of  Commander,  Naval  Sea  Systems  Command: 

Our  Common  mission  is  to  develop,  deliver  and  maintain  ships  and  systems  on 
time  and  on  cost  for  the  Navy.  To  build  an  affordable  future  fleet,  we  will  focus 
on  reducing  acquisition  costs,  including  applying  more  risk-based  decisions  to 
specifications  and  requirements.  (McCoy,  2008) 

The  United  States  Navy  is  living  and  functioning  in  an  era  of  ever-expanding  technology, 
more  stringent  requirements,  and  a  growing  need  for  more  ships  and  resources,  all  while 
working  with  a  limited  budget.  These  factors  all  lead  to  inherent  cost  growth  in  the  projects  that 
are  developed  to  provide  the  fleet  with  the  capabilities  it  needs.  In  order  for  the  United  States 
Navy  to  acquire  and  provide  a  full, 

state-of-the-art,  313-ship  Navy  by  2020,  as  stated  in  the  fiscal  year  (FY)  2007  plan  (Department 
of  the  Navy,  2006)  it  is  imperative  that  methods  allowing  full  capitalization  of  each  dollar  spent 
by  the  Navy  are  developed  and  implemented. 

In  February  2006,  the  United  States  Navy  presented  its  FY2007  plan,  which  outlines  the 
objective  of  increasing  the  current  285-ship  fleet  to  313  ships  by  2020  (Department  of  the  Navy, 
2006).  By  2008,  the  Navy  increased  the  estimate  of  its  annual  cost  for  the  30-year  plan  by 
about  44%  in  real  terms,  but  it  is  still  approximately  7%  less  than  independent  cost  estimates 
conducted  by  the  Congressional  Budget  Office  (O’Rourke,  2008).  This  increase  in  estimated 
cost  poses  a  problem  for  the  overall  funding  of  the  shipbuilding  strategy  proposed  by  the 
Secretary  of  the  Navy.  The  credibility  of  the  Navy’s  estimates  and  the  ability  to  fund  its 
shipbuilding  plans  have  been  questioned  by  Congress  and  industry  (Cavas,  2008). 

In  order  to  adequately  generate  the  funds  to  implement  the  30-year  plan  of  future  ships 
and  capabilities,  the  Navy  must  explore  different  options  for  cost  savings.  Fundamental  to  the 
success  of  complex  projects,  such  as  naval  vessel  construction,  is  the  ability  to  control, 
manage,  and  communicate  the  status  of  the  risk  reduction  effort  throughout  the  development 
and  production  cycles  (Kujawski  &  Angelis,  2007).  It  is  recognized  that  the  Navy  and 
shipbuilding  industry  need  to  change  their  technical  and  business  shipbuilding  strategies  in 
order  to  achieve  the  goal  of  a  future  fleet  that  balances  both  capability  and  affordability.  Cost 
risk  assessment  and  analysis  is  one  tool  that  can  be  utilized  to  help  recapitalize  costs  used  in 
the  ship  acquisition  and  building  process. 


DEFENSE  ACQUISITION  IN  TRANSITION 


-115- 


A.  Background 

Risk  analysis  is  an  important  component  of  the  cost  analysis  of  new  vessels  because 
actual  costs  will  always  have  a  probability  of  differing  from  the  estimate.  Several  reasons 
account  for  the  difference  between  the  estimate  and  actual  cost,  which  can  include  lack  of 
knowledge  about  the  future,  errors  associated  with  assumptions  and  cost-estimating  equations, 
historical  data  inconsistencies,  and  factors  considered  in  making  the  estimate.  The  overall 
purpose  of  risk  analysis  is  to  quantify  the  potential  for  error  (GAO,  2007).  In  the  case  of  a  cost 
estimate,  it  is  the  probability  that  the  actual  cost  will  exceed  the  cost  estimate  or  the  budget. 

This  cost  estimate  allows  for  the  assessment  of  risk  of  a  given  program. 

Cost  overruns  and  growth  are  an  enduring  problem  that  is  not  new  to  the  Navy.  A  1939 
inquiry  from  Secretary  of  the  Navy  Ray  Spear  asks  the  question,  “Why  do  naval  vessels  cost  so 
much?”  Answers  to  this  inquiry  include  increased  progress  in  marine  engineering  and  naval 
construction,  increased  horsepower  in  shipbuilding,  improved  quality  of  building  materials, 
inflation,  and  the  practice  of  paying  full  price  for  the  best  you  can  buy  naturally  increases  costs. 
Spear  (1939)  states  that,  “care  must  be  taken  in  approving  estimates  to  make  sure  that  they  are 
reasonable  and  held  to  in  the  cost  of  production.  When  contracts  are  negotiated,  the  question 
of  costs  should  be  investigated  and  a  detailed  knowledge  of  approximate  costs  obtained.”  Just 
as  cost  estimation  was  recognized  by  the  Secretary  of  the  Navy  in  1939,  it  is  recognized  by 
today’s  Navy  leadership  as  an  integral  part  of  the  ship  acquisition  process. 

Risk  analysis  and  management  can  be  used  to  help  program  managers  more  effectively 
make  acquisition  decisions  and  allocate  their  resources  by  allowing  for  a  better  understanding  of 
program  risks.  Risk  management  is  a  continuous  process  in  the  acquisition  and  development  of 
naval  vessels. 

The  Naval  Sea  Systems  Command,  Cost  Engineering  and  Industrial  Analysis  (NAVSEA 
05C)  introduced  Cost  Risk  Analysis  (CRA)  into  the  Navy’s  PR09  Planning,  Programming, 
Budgeting,  and  Execution  System  (PPBES)  to  help  assess  vessel  costs  in  terms  of  quantifiable 
risk.  Cost  Risk  Assessment  is  defined  as  the  process  of  quantifying  the  uncertainties 
associated  with  major  acquisition  programs.  It  therefore  allows  for  informed  decisions  with  an 
estimated  level  of  confidence  (McCarthy,  2008). 

One  of  the  key  objectives  of  CRA  is  to  enable  better  risk  management,  which  will 
simultaneously  reduce  program  costs  and  increase  the  probability  of  success.  Cost  estimating 
is  recognized  by  NAVSEA  05C  as  an  essential  element  of  effective  program  management, 
required  for  realistic  program  planning  and  decision-making.  Risk  analysis  is  important  because 
the  previous  methodology  of  using  point  estimates  is  “precisely  wrong”  (Deegan,  2007).  Risk 
cannot  be  assessed  with  a  point  estimate,  as  it  represents  a  single  value  that  serves  as  a  best 
guess  for  the  parameter  to  be  defined.  Decision-makers  may  not  be  able  to  completely 
understand  the  influence  of  different  variables  on  cost  with  the  use  of  a  point  estimate. 
Conversely,  the  use  of  risk  analysis  allows  the  decision-maker  to  utilize  his  or  her  acquisition 
experience,  while  quantifying  the  qualitative  aspects  of  acquisition  scenarios. 

Point  estimates  are  not  an  accurate  method  for  predicting  costs  in  shipbuilding  because 
they  do  not  properly  account  for  problems  that  may  be  encountered  in  the  acquisition  process, 
as  described  above.  They  may  be  either  overly  optimistic  or  overly  pessimistic.  Optimistic  point 
estimates  ignore  the  potential  risk  and  uncertainty  in  a  project,  which  is  necessary  for 
management  to  make  informed  decisions.  Immature  technology,  uncertain  product  design, 
schedule  problems,  and  unforeseen  events  all  have  risk  associated  with  their  end  product.  Risk 
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analysis  is  necessary  in  order  to  incorporate  the  effect  of  risk  into  the  overall  cost.  Pessimistic 
point  estimates  assume  worst  scenarios  and  unlikely  high  costs.  Quantitative  risk  analysis 
allows  the  cost  estimator  to  assign  a  realistic  range  of  costs  around  a  point  estimate,  which 
provides  decision-makers  with  a  level  of  confidence  in  achieving  a  credible  cost. 

The  NAVSEA  Cost-estimating  Process  is  comprised  of  three  parts,  which  are  further 
divided  into  12  tasks.  The  three  parts  consist  of  Develop  Approach,  Perform  Estimate,  and  Brief 
Results.  Figure  1  shows  the  breakdown  of  the  12  tasks  within  the  three  parts.  This  paper 
focuses  on  the  Develop  Approach  and  Perform  Estimate  parts  of  the  cost-estimating  process. 


RES,J^ 
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Task  1.  Initiate  Estimate 
TaskZ  Develop  Technical  Baseline 
Task  3.  Obtain  WBS 
Task  4.  Establish  Estimate  Assumptions 
Task  5.  Select  Cost  Estimating  Methods 
&  Tools 

Task  6.  Collect  Data 


Part  1 


Task  7.  Run  Model  8c  Generate  Point 
Estimate 

Task  8.  Conduct  Cost  Risk  Analysis  & 
Incorporate  into  Estimate 
Task  9.  Conduct  Preliminary  Estimate 
Review 


Task  10.  Produce  Final  Estimate 
Task  1 1 .  Document  Estimate 
Task  1Z  Brief  Results 


Part  2 


Part  3 


Figure  1.  NAVSEA  12-Step  Cost-estimating  Process 

(NAVSEA,  2005) 


Data  collection  is  a  task  within  the  Develop  Approach  part  of  cost  estimation  and  can  be 
regarded  as  the  most  important  part  of  risk  analysis.  Bad  data  will  produce  bad  results, 
regardless  of  the  subsequent  analysis.  Data  elicitation  is  often  done  ad  hoc;  however,  several 
reliable  methods  and  sources  are  available  for  data  collection.  Data  quality  is  critical  to  the 
success  of  the  analysis  and  plays  a  significant  part  in  the  results  generated  for  cost  estimation. 
This  paper  will  discuss  improved  methods  for  data  collection  in  order  to  obtain  more  reliable  and 
standardized  data  from  subject  matter  experts  (SMEs). 

Risk  analysts  use  probability  distributions  rather  than  point  estimates  to  represent  the 
possible  outcomes  of  an  event.  There  is  a  significant  difference  between  a  point  estimate  and  a 
distribution,  in  that  the  distribution  provides  the  full  range  of  values  with  their  associated 
probabilities,  while  the  point  estimate  presents  a  single  value.  This  allows  program 
management  to  make  budget  decisions,  based  on  desired  confidence  levels.  Quality  may  differ, 
based  on  the  method  of  collection.  Two  methods  commonly  used  for  data  collection  include 
database  queries  and  interviews  of  SMEs  or  stakeholders  (Deegan  &  Fields,  2007).  This  paper 
analyzes  the  current  NAVSEA  05C  Cruiser  (CG(X))  probabilistic  cost  model  including  data 
elicitation. 

The  direct  fractile  assessment  (DFA)  method  provides  one  of  the  most  reliable  and  least 
bias-prone  procedures  for  eliciting  uncertain  quantities  from  SMEs  (Kujawski,  Alvaro  & 

Edwards,  2004).  Data  elicitation  from  SMEs  is  innately  uncertain;  three  findings  from 
psychological  experiments  conducted  by  Alpert  and  Raiffia  (1982)  are: 


DEFENSE  ACQUISITION  IN  TRANSITION 


-117- 


■  A  systematic  bias  toward  overconfidence  is  common. 

■  Extreme  value  judgment  is  poor. 

■  Maximum  and  minimum  values  are  vague  terms.  What  do  these  terms  really  mean? 

Based  on  the  findings  of  Alpert  and  Raiffia  (1982),  Kujawski  et  al.  (2004)  propose  the 
following  guidelines  for  data  elicitation: 

■  Ask  SMEs  to  provide  10th,  50th,  and  90th  percentiles  values  for  cost  elements. 

Avoid  extreme  values,  abstract  measures  such  as  the  mean  or  standard  deviation,  or 
specific  distribution  functions.  Allow  for  discussion  and  education  of  the  SME  in 
terms  of  bias  when  giving  data  figures. 

■  Calibrate  each  set  of  percentiles  to  reflect  individual  and  project  specific 
considerations,  both  pessimistic  and  optimistic.  For  estimates  that  might  be  overly 
optimistic,  a  cost  analyst  might  choose  to  shift  a  90th  percentile  value  to  perhaps 
80th  or  75th  percentiles. 

Tasks  involved  in  the  Performing  Estimate  depicted  in  Figure  1  are  running  the  model 
and  generating  a  point  estimate  or  probability  distribution,  conducting  a  cost  risk  analysis,  and 
conducting  a  preliminary  estimate  review. 

Traditionally,  triangular  distributions  have  been  used  in  cost  estimation  models  because 
of  the  simplicity  in  entering  the  required  data.  The  triangular  distribution  requires  minimum  or 
low,  most-likely,  and  high  or  maximum  values.  Other  commonly  used  distributions  include 
normal,  lognormal,  and  uniform.  Table  1  lists  eight  of  the  most  common  probability  distributions 
used  for  cost  estimation  and  uncertainty  analysis.  This  paper  investigates  different  methods  for 
data  elicitation  and  selecting  appropriate  distributions.  The  effects  of  using  different 
distributions  on  cost  risk  are  evaluated  and  identified. 
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Distribution 


Typical  application 


Description  Shape 

Lognormal  A  continuous  distribution  positively  prc  s  at,  i.ty  To  characterize  uncertainty  in 

skewed  with  a  limitless  upper  bound  nonlinear  cost  estimating 

and  known  lower  bound;  skewed  to  relationships, 

the  right  to  reflect  the  tendency  toward 
higher  cost. 

Values 


Normal  Used  for  outcomes  likely  to  occur  on 

either  side  of  the  average  value; 
symmetric  and  continuous,  allowing 
for  negative  costs  and  durations.  In  a 
normal  distribution,  about  68%  of  the 
values  fall  within  one  standard 
deviation  of  the  mean. 

Beta  Similar  to  normal  distribution  but  does  Pmoac  iity  To  capture  outcomes  biased 

not  allow  for  negative  cost  or  duration,  toward  the  tail  ends  of  a  range; 

this  continuous  distribution  can  be  often  used  with  engineering 

symmetric  or  skewed.  data  or  analogy  estimates. 


To  assess  uncertainty  with 
cost  estimating  methods;  the 
standard  deviation  or  standard 
error  of  the  estimate  is  used  to 
determine  dispersion. 


Triangular  Characterized  by  three  points — most  PrctaDi  ty  To  express  technical 

likely,  pessimistic,  and  optimistic  uncertainty,  because  it  works 

values — can  be  skewed  or  symmetric  for  any  system  architecture  or 

and  is  easy  to  understand  because  it  design;  also  used  to  determine 

is  intuitive.  One  drawback  is  the  schedule  uncertainty, 

absoluteness  of  the  end  points.  values 


Uniform  Has  no  peaks  because  all  values,  Pite.it,  i,ty  With  engineering  data  or 

including  highest  and  lowest  possible  analogy  estimates, 

values,  are  equally  likely. 


Weibull  Versatile,  able  to  take  on  the  Ptcbac  i  ty  In  life  data  and  reliability 

characteristics  of  other  distributions,  analysis  because  it  can  mimic 

based  on  the  value  of  the  shape  other  distributions  and  its 

parameter  "b"— e.g.,  Rayleigh  and  objective  relationship  to 

exponential  distributions  can  be  reliability  modeling, 

derived  from  it.'  v  . 


Source:  DOD,  NASA,  SCEA,  and  Industry. 

The  Rayleigh  and  exponential  distributions  are  a  class  of  continuous  probability  distribution. 


Table  1.  Common  Probability  Distributions  Used  in  Cost-estimating  Uncertainty  Analysis 

(GAO,  2007) 


The  Money  Allocated  is  Money  Spent  (MAIMS)  principle  is  based  on  Parkinson’s  Law, 
where  “Work  expands  to  fill  the  time  allotted”  and  “padding  schedule  estimates  directly 
contribute  to  cost  overruns”  (Augustine,  1997).  In  other  words,  it  suggests  that  there  will  be  no 
cost  underruns,  and  that  the  project  will  come  in  at  or  above  the  cost  to  which  it  is  funded. 
Implementing  the  MAIMS  principle  in  Monte  Carlo  simulations  modifies  the  basic  probability 
distribution  functions  (PDF)  by  setting  any  value  less  than  the  money  allocation  point  equal  to 
that  money  allocation  value.  There  will  be  no  costs  associated  with  a  value  less  than  this 
money  allocation  point.  Utilizing  the  MAIMS  principle,  the  PDFs  are  modified  to  include  a  spike 
or  delta  function  at  an  arbitrary  point,  which  is  assumed  to  be  the  “money  allocation  point,” 
corresponding  to  the  dollar  amount  allocated  to  the  program  manager  for  the  project  and/or 
project  cost  elements. 

Correlation  effects  between  elements  are  analyzed.  Correlation  accounts  for 
interrelationships  between  cost  elements.  Data  elements  can  either  be  negatively,  neutrally,  or 
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positively  correlated  and  can  either  exist  among  cost  elements  within  a  subsystem  or  between 
elements  in  different  subsystems.  For  example,  take  into  consideration  the  elements  of  a  ship. 
Positive  correlation  arises  when  increases  in  weight,  size,  and  number  of  weapons  systems 
onboard  result  in  an  increase  in  acquisition  and  shipbuilding  costs.  An  increase  in  the 
complexity  of  a  weapon  system  further  forces  an  increase  in  cost  of  other  systems  such  as 
power,  cooling,  control.  Analysis  would  be  greatly  simplified  if  analysts  could  assume  that  all 
elements  are  independent  or  that  all  elements  are  dependent.  Since  neither  statement  is  true, 
correct  correlation  between  elements  is  necessary  to  provide  the  most  accurate  representation 
of  cost. 


Many  software  programs  are  available  for  cost  risk  analysis.  This  paper  uses  Crystal 
Ball®  as  an  add-in  to  Microsoft  Excel  ,  because  of  its  ease  of  use  and  because  it  is  the  current 
program  used  by  NAVSEA  05C.  Crystal  Ball®  generates  the  Monte  Carlo  simulations  that 
become  the  backbone  of  the  cost  risk  analysis.  A  Monte  Carlo  simulation  calculates  multiple 
scenarios  of  a  model  by  repetitively  sampling  values  from  the  input  variable  distributions  for 
each  uncertain  variable  and  then  calculates  the  result.  The  resulting  cost  distributions  from 
Crystal  Ball"  provide  the  decision-maker  with  powerful  cost  risk  information. 

A  program  built  on  a  solid  foundation  of  accurate  cost  estimating  that  effectively 
considers  risks,  combined  with  strong  systems  engineering  and  program  management,  gives 
the  program  a  greater  chance  of  success. 

B.  Purpose 

The  purpose  of  this  paper  is  to  analyze  the  probabilistic  cost  analysis  approach  that 
NAVSEA’s  Cost  Engineering  and  Industrial  Analysis  division  (SEA  05C)  currently  uses  to 
predict  new  naval  vessel  construction  costs  and  to  develop  a  method  that  better  predicts  the 
ultimate  cost  risk.  This  paper  uses  data  collected  from  analysis  of  the  CG(X)  class  ship  cost 
model.  The  model  used  to  determine  cost  is  reviewed  and  analyzed  to  determine  what  factors 
should  be  considered  to  produce  more  realistic  cost  estimates. 

II.  Revised  Cost  Risk  Analysis 

A.  Introduction 

The  NAVSEA  05C’s  cost  analysis  model  of  CG(X)  encompasses  all  aspects  of  cost  for 
the  entire  fleet  including  inflation  and  profit  in  a  63  worksheet  Excel®  workbook.  The  data  used 
in  the  NAVSEA  model  were  acquired  from  SME  inquiry  using  three-point  estimates  of  high, 
most-likely,  and  low  values.  In  order  to  ensure  the  continued  acquisition  of  CG(X),  it  is 
important  that  realistic  cost  risk  analysis  be  performed  so  that  program  managers  can  make 
informed  decisions. 

This  chapter  presents  an  approach  to  improve  on  the  model  that  NAVSEA  05C  has 
provided  for  CG(X).  The  focus  is  strictly  on  the  methodology  used  in  the  cost  analysis  of  the 
electronics  suite  of  CG(X)  cost  model  and  cost  uncertainties  associated  with  engineering  and 
manufacturing  of  the  lead  vessel.  Nine  systems  make  up  the  electronics  suite: 

■  Radar  suite,  which  consists  of  the  following  subsystems:  X-band,  S-band,  Cooling, 
and  Power 

■  ExComm — External  Communications 
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■  TSCE — Total  Ship  Computing  Environment 

■  IUSW — Integrated  Undersea  Warfare 

■  EW-IW — Electronic  Warfare-Information  Warfare 

■  EO-IR — Electro  Optical-Infrared 

■  IFF — Identification,  Friend  or  Foe 

■  MS  EI&T  (SS  Only) — Mission  Systems  Engineering,  Integration,  and  Testing  (Ship 
Systems  Only) 

■  MS  EI&T  (CS  Only) — Mission  Systems  Engineering,  Integration,  and  Testing 
(Combat  Systems  Only) 

The  electronics  suite  cost  is  determined  with  the  following  two  equations  that  treat  each 
of  the  cost  elements  as  a  random  variable  (RV).  The  costs  in  bold  represent  composite  of  the 
costs  in  regular  font,  which  Crystal  Ball®  refers  to  as  forecasts  and  assumptions: 

COST(Electronics  Suite)  =  COST(Radar  Suite)  +  COST(ExComm)  +  COST(TSCE)  + 
COST(IUSW)  +  COST(EW-IW)  +  COST(EO-IR)  +  COST(IFF)  +  COST(MS  EI&T  (SS 
Only))  +  COST(MS  EI&T  (CS  Only)), 

Cost(Radar  Suite)  =  Cost(X-band)  +  Cost(S-band)  +  Cost(Cooling)  +  Cost(Power). 


The  steps  of  analysis  for  the  CG(X)  model  are  as  follows: 

1 .  Analyze  the  cost  factors  used  by  NAVSEA  05C  to  develop  the  electronics  suite  cost. 

2.  Analyze  the  PDFs  used  for  the  electronics  cost  elements. 

3.  Identify  what  data  elicitation  methods  were  employed. 

4.  Determine  if  correlation  factors  were  used  in  the  cost  analysis. 

5.  Develop  cost  factors  to  be  modeled  for  cost  realism. 

6.  Decide  which  PDFs  to  use  for  greater  fidelity. 

7.  Develop  an  improved  cost  risk  model  that  includes  realistic  correlation  factors; 
credible  PDFs,  including  MAIMS  influences;  and  SME  biases. 

B.  Review  Development  of  Cost  Factors 

1.  Data  Elicitation  Methods 

The  data  elicitation  methods  used  by  NAVSEA  05C  cost  analysts  are  not  well 
documented.  It  is  clear  that  the  engineering  and  expert  judgment  of  SMEs  is  heavily  relied  on 
for  the  assessment  of  uncertain  cost  elements  associated  with  new  designs.  This  is  an  area 
where  the  use  of  improved  methods  can  dramatically  improve  the  quality  of  data  that  is  used  in 
the  computation  of  the  cost  risk  model.  Subjective  assessments  to  obtain  data  have  been 
identified  as  a  critical  source  of  uncertainty  in  probabilistic  risk  analyses  (Keeney  &  von 
Winterfeld,  1991).  Kujawski  et  al.  (2004)  discuss  the  use  of  the  DFA  method  for  data  elicitation 
and  how  this  ties  in  with  distribution  choice,  to  provide  the  most  realistic  cost  assessment. 
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DFA  has  been  found  to  provide  one  of  the  most  consistent  and  least  bias-prone  methods 
for  eliciting  uncertain  quantities  (Alpert  &  Raiffa,  1982).  In  their  research,  people  were  asked  to 
consider  uncertain  quantities  by  providing  values  in  terms  of  percentiles  or  fractiles.  The 
findings  indicated: 

■  There  is  a  systematic  bias  toward  overconfidence  in  estimates.  The  subjective 
probability  distributions  were  too  narrow.  Usually,  33%  instead  of  50%  of  the  actual 
values  fell  within  the  0.25  to  0.75  fractiles. 

■  Extreme  value  judgment  is  even  worse:  20%,  rather  than  2%,  of  the  actual  values  fell 
outside  the  0.01  and  0.99  fractiles. 

■  What  is  the  meaning  of  minimum  and  maximum  values?  Defining  these  terms,  so 
that  they  are  universal,  is  difficult. 

Kujawski  et  al.  (2004)  further  suggest  using  experts  to  provide  the  10th,  50th,  and  90th 
percentile  values,  as  these  may  be  easier  to  assess  than  extreme  values  of  maximum  and 
minimum.  They  recommend  avoiding  asking  for  extreme  values,  abstract  values  such  as  the 
mean  or  standard  deviation,  or  other  specific  distribution  functions.  If  the  analyst  does  not  fully 
understand  the  background  of  the  questions  being  asked,  or  if  he  or  she  does  not  fully 
understand  the  behavior  of  the  system  and  associated  data,  obtaining  discrete  values  will  be 
near  impossible. 

Education  also  plays  an  important  role  in  the  quality  of  the  data  provided  by  the  SMEs 
for  analysis.  The  understanding  of  bias  and  its  role  in  affecting  data  elicitation  is  important.  In  a 
presentation  to  the  Navy  Cost  Analysis  Symposium,  Fields  and  Popp  (2007)  stress  the 
importance  of  several  lessons  learned  on  risk.  One  of  the  most  interesting  of  these  lessons 
learned  is  the  importance  of  training.  They  indicate  that  although  NAVSEA  and  its  technical 
community  have  a  broad  cross  section  of  educational  backgrounds  and  experience,  not 
everyone  has  experience  in  simulation  and  statistics.  The  SME  for  a  particular  electronics  suite 
component  is  probably  not  an  expert  in  probability  and  statistics,  and  because  of  this,  tends  to 
give  biased  answers  to  the  cost  analysis.  The  distributions  formed  from  the  biased  data  have 
been  found  to  be  particularly  narrow  and  centered  on  a  given  point  estimate,  while  the  extreme 
values  are  very  rarely  taken  into  account,  for  reasons  described  above. 

Education  of  the  SMEs  while  conducting  data  elicitation  is  important,  so  that  the  experts 
have  a  better  understanding  of  what  data  is  required  and  how  it  is  going  to  be  utilized.  This 
training  needs  to  be  continually  refreshed  due  to  the  high  turnover  rate  of  personnel,  whether 
they  be  military  or  civilian,  and  also  because  of  improving  methods  for  cost  analysis.  An 
adequate  training  plan  for  both  the  cost  analysts  and  the  SMEs  providing  data  will  ultimately 
result  in  better  data  acquisition  for  cost  analysis. 

In  this  paper,  the  use  of  DFA  is  simulated  though  the  use  of  Weibull  probability 
distributions  because  no  new  data  elicitation  was  conducted.  The  differences  between  the 
distributions  using  identical  values  for  10th,  50th,  and  90th  percentiles  versus  the  20th,  50th, 
and  80th  percentiles  illustrates  data  elicitation  that  is  optimistic  versus  pessimistic.  The 
resulting  cost  associated  with  each  of  the  two  distributions  shows  how  dramatic  the  effects  of 
slightly  different  parameters  can  have  on  the  estimated  cost. 
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2. 


Choice  and  Development  of  Probability  Distribution  Functions 


Kujawski  et  al.  (2004)  emphasize  the  importance  of  realistically  modeling  cost 
uncertainties  through  the  appropriate  choice  of  probability  distribution  by  meeting  the  following 
criteria: 

■  Capable  of  fitting  three  arbitrary  percentiles. 

■  A  finite  lower  range. 

■  An  infinite  upper  range  with  reasonable  behavior. 

■  Physically  meaningful  and  easy  to  estimate  parameters. 

Three  types  of  PDFs  are  developed  and  modeled  for  this  paper,  with  the  goal  of  finding 
a  realistic  and  flexible  probability  distribution.  Uncertainty  for  each  cost  element  in  the  cost 
model  is  represented  using  the  same  type  of  PDF  with  different  parameters  (based  on  NAVSEA 
data).  First,  a  triangular  PDF  that  uses  low,  most-likely,  and  high  values  for  its  parameters  is 
developed.  A  lognormal  PDF  that  uses  the  mean  and  standard  deviation  from  the  triangular 
PDF  as  its  parameters  is  the  second  distribution.  The  third  PDF  is  a  three-parameter  Weibull 
distribution  based  on  the  low,  most-likely,  and  high  values  of  the  triangular  PDF  provided  by 
NAVSEA  05C.  The  low  and  high  values  are  calculated  by  multiplying  the  low  and  high 
percentages  obtained  with  the  most-likely  value.  Two  Weibull  distributions  are  created.  One  of 
the  Weibull  distributions  uses  the  data  as  input  for  the  10th,  50th,  and  90th,  percentiles,  while 
the  other  is  more  pessimistic  and  uses  these  values  for  the  20th,  50th,  and  80th  percentiles. 

For  consistency,  the  triangular  distribution  is  used  to  determine  the  50th  percentile.  The  low, 
50th  percentile,  and  high  values  are  substituted  for  the  20th,  50th,  and  80th  percentiles  and  for 
the  10th,  50th,  and  90th,  for  each  three-parameter  Weibull  distribution,  respectively. 

a.  Triangular  Probability  Distribution  Function 

The  parameters  used  to  develop  the  triangular  PDF  are  the  low,  most-likely,  and  high 
values  from  the  Mission  Systems  Risk  Assessment  worksheet  of  the  CG(X)  model.  The 
determination  of  the  high  and  low  percentages  for  cost  values  in  the  Mission  Systems  Risk 
Assessment  worksheet  were  figures  given  to  NAVSEA  05C  cost  analysts  by  SMEs  from  the 
NSWC  Dahlgren.  These  percentages  are  based  on  historical  database  values  and  inquiry  of 
the  SME  for  an  opinion  about  what  the  low  and  high  values  would  be,  based  on  the  most-likely 
values  obtained  from  the  historical  databases.  In  this  case,  data  elicitation  plays  a  big  part  in 
the  reliability  of  the  data  used  in  the  model,  which  is  to  be  described  in  more  depth  in  Section 
IV.B.1 . 


The  triangular  distribution  is  not  a  good  predictor  of  high  and  low  costs  because  it  uses 
the  low  and  high  values  as  extreme  values  for  the  end  points.  There  is  no  allowance  for  costs 
above  or  below  the  input  values.  It  has  been  argued  that  a  triangular  distribution  can  lead  to 
either  underestimates  or  overestimates.  Graves  (2001)  states  that  underestimates  are  likely 
due  to  the  finite  upper  limit  of  the  distribution.  Moran  (1999)  believes  that  overestimates  happen 
because  of  the  distribution’s  inability  to  portray  the  expert’s  confidence  level  of  achieving  the 
most-likely  value  and/or  knowledge  of  the  shape  of  the  distribution.  The  triangular  distribution  is 
assigned  a  very  low  score  for  criteria  (i)  and  (iii)  (Kujawski  et  al.,  2004)  and  is  not  the  chosen 
distribution  to  represent  cost  in  the  model  for  this  paper. 
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b.  Lognormal  Probability  Distribution  Function 

The  lognormal  PDF  is  created  with  the  mean  and  standard  deviation  parameters  taken 
from  the  triangular  distribution.  Characteristics  of  a  lognormal  distribution  include  being 
positively  skewed  with  a  limitless  upper  bound  and  known  lower  bound.  This  distribution  is 
assigned  an  acceptable  score  for  criteria  (iii),  but  a  low  score  for  (i),  due  to  the  always  positively 
skewed  nature  of  the  distribution.  The  lognormal  distribution  results  in  a  cost  profile  that  closely 
follows  with  the  triangular  distribution  and  is  one  of  the  narrowest  profiles  modeled.  A  lognormal 
PDF  has  been  associated  with  providing  unreasonably  high  probabilities  at  high  values,  due  to 
the  relatively  slow  falloff  to  the  right.  For  this  reason,  it  gets  an  acceptable  score  for  the  criteria 
(iii)  but  scores  low  on  the  criteria  (i)  because  of  its  always  positively  skewed  characteristic 
(Kujawski  et  al. ,  2004). 

c.  Weibull  Probability  Distribution  Function  (10%,  50%,  90%) 

The  three-parameter  Weibull  distribution  is  characterized  by  being  flexible  and  able  to 
assume  a  wide  variety  of  shapes,  while  also  being  open-ended.  Because  of  its  flexible  profile 
and  ability  to  mimic  other  distributions,  it  scores  high  on  all  criteria.  This  paper  models  one  of 
the  three-parameter  Weibull  distributions  with  the  10th,  50th,  and  90th  percentile  values  for  cost. 
The  parameters  of  10th,  50th,  and  90th  percentiles  are  chosen  to  simulate  a  cost  environment 
that  allows  for  some  cost  flexibility  on  the  upper  and  lower  limits  rather  than  making  them 
extreme  as  in  the  triangular  distribution.  Although  this  1 0%  change  on  either  side  of  the 
distribution  seems  large,  it  actually  represents  a  fairly  optimistic  assessment  of  cost.  This  model 
is  best  for  a  situation  in  which  the  data  obtained  for  the  model  is  very  reliable. 

d.  Weibull  Probability  Distribution  Function  (20%,  50%,  80%) 

The  three-parameter  Weibull  distribution  using  the  20th,  50th,  and  80th  percentiles  for 
distribution  parameters  is  intended  to  correct  or  account  for  the  overly  optimistic  biases 
discussed  in  Section  2  above.  Systems  that  are  new  and  untested  have  a  certain  amount  of 
uncertainty  inherent  in  their  acquisition,  and  most  cost  assessments  made  on  their  components 
are  based  on  past  history  if  components  are  being  reproduced,  or  a  best  estimate  for  new 
systems  and  their  components.  SMEs  are  naturally  optimistic  about  their  systems  and  have 
been  shown  to  give  cost  estimates  that  are  overconfident,  resulting  in  probability  distributions 
that  do  not  accurately  reflect  the  possible  range  of  costs  (Kujawski  et  al.,  2004). 

Much  of  the  data  for  the  CG(X)  electronics  suite  is  the  result  of  SME  inquiry,  which 
explains  why  the  Weibull  distribution  using  20%,  50%,  and  80%  parameters  is  chosen  to  model 
costs  for  the  electronics  suite  components  in  this  paper. 

e.  Cost  Comparisons  with  the  Different  Probability  Distributions 

Figure  2  is  the  Excel®  overlay  created  with  Crystal  Ball®  that  shows  of  a  10,000-run 
Monte  Carlo  simulation  for  the  triangular,  lognormal,  Weibull  (10%,  50%,  90%)  and  Weibull 
(20%,  50%,  80%)  distributions,  representing  the  electronics  suite  cost  of  the  CG(X).  Figure  10 
is  the  cumulative  probability  distribution  derived  from  the  PDF  shown  in  Figure  9.  The  triangular 
and  lognormal  distributions  are  very  similar  in  both  the  probability  distribution  and  cumulative 
frequency  functions,  which  is  expected.  Since  the  lognormal  distribution  uses  the  mean  and 
standard  deviation  from  the  triangular  distribution  as  its  parameters,  the  end  result  should  be 
very  similar.  Both  the  triangular  and  lognormal  functions  show  a  distinct  peak  and  sharp  falloff 
at  both  the  lower  and  upper  bounds.  This  behavior  does  not  realistically  model  the  electronics 
suite  cost  because  of  the  sharp  peak  with  sharp  falloff. 
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Data  Elicitation  and  Distribtuion  Selection  Effects 
for  CG(X)  Electronics  Cost 
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Figure  2.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs  Showing  the  Effects  of  Distribution 
Choices  on  the  Cost  Probability  Distributions 
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Figure  3.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs  Showing  the  Effects  of  Distribution 
Choices  on  the  Cost  Cumulative  Distribution  Functions 

The  Weibull  (10%,  50%,  90%)  distribution  shows  a  broader  cost  range  for  the  given 
probability  brackets.  The  tapering  lower  and  upper  bounds  in  comparison  to  the  triangular  and 
lognormal  distribution  represent  a  more  likely  cost  outcome.  The  Weibull  (20%,  50%,  80%) 
shows  an  even  larger  cost  range,  which  makes  sense  because  this  distribution  is  supposed  to 
model  a  more  pessimistic  view  of  cost.  Both  of  these  distributions  are  associated  with  higher 
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costs  as  the  probability  of  the  cost  increases.  It  is  important  to  note  the  difference  between  the 
optimistic  and  pessimistic  Weibull  distributions  in  Figure  3.  For  each,  the  cost  increases  with  an 
increase  in  probability,  but  it  is  clear  that  the  model’s  results  indicate  that  the  cost  risk  is 
significantly  higher  using  the  pessimistic  Weibull  distribution. 


3.  Correlation  Effects 

Correlation  effects  are  potentially  important  in  modeling  appropriate  cost  relationships 
between  different  elements  of  systems  and  are  not  conducted  enough  in  current  cost  analysis 
models  (Book,  2001).  Trends  with  correlation  tend  to  lean  toward  perfect  correlation  because  of 
simplicity.  Perfect  correlation  helps  to  widen  the  range  of  outputs  in  the  distribution  functions, 
but  this  may  not  be  an  accurate  or  reliable  representation.  Reasonable  correlation  coefficients 
may  provide  more  realistic  and  credible  estimates  of  project  costs,  rather  than  assuming  perfect 
or  zero  correlation.  Assessing  correlation  coefficients  is  a  difficult  problem.  A  need  exists  for 
the  investigation  and  development  of  a  realistic  and  practical  model  to  account  for 
interrelationships  between  cost  elements. 

Two  types  of  correlations  are  modeled  in  this  paper: 

■  Correlations  among  cost  elements  within  the  radar  suite.  The  radar  suite  includes 
elements  of  X-band,  S-band,  Cooling,  and  Power.  Dependencies  among  these 
components  are  mainly  from  subsystem  characteristics  such  as  complexity. 

■  Correlations  among  cost  elements  in  the  entire  electronics  suite.  Dependencies 
among  these  cost  elements  occur  from  the  programmatic  and  organizational 
considerations  common  to  all  cost  elements  that  are  a  part  of  the  same  project 
(Kujawski  et  al. ,  2004). 

There  are  two  types  of  correlations:  Pearson  and  Spearman.  Pearson  correlation 
coefficient  determines  the  degree  of  linearity  between  two  random  variables,  while  Spearman 
rank  order  correlation  coefficients  measure  monotonicity.  Correlation  among  cost  elements  in 
the  electronics  suite  is  modeled  with  the  use  of  the  Correlation  Matrix  function  in  Crystal  Ball®. 
Crystal  Ball®  uses  rank  correlations  to  correlate  assumptions.  This  means  that  the  values  are 
not  changed,  but  they  are  rearranged  to  produce  the  desired  correlation.  Rank  correlation 
eliminates  the  need  to  explicitly  model  the  dependence  between  the  cost  elements.  Garvey 
(2000)  advocates  the  use  of  Pearson’s  correlation.  However,  given  the  limited  information,  rank 
correlations  offer  the  advantage  of  accounting  for  correlations  independent  of  explicit  distribution 
and  dependency  models.  The  use  of  Monte  Carlo  simulations  generates  the  full  PDF  rather 
than  simply  expected  value  and  variance. 

This  paper  uses  three  sets  of  two  correlation  coefficients  to  model  the  correlation 
between  the  radar  suite  elements  and  the  rest  of  the  electronics  suite  components.  The  first  set 
models  the  distributions  with  correlation  coefficients  of  0.5  for  the  radar  suite  elements  and  0.5 
for  the  entire  electronics  suite  elements.  The  second  set  of  correlation  coefficients  is  0.5  and 
0.2.  The  third  set  uses  correlation  coefficients  of  0.8  and  0.5. 

Figure  4  is  an  overlay  of  the  different  probability  distributions  for  the  electronics  suite 
cost,  produced  by  using  the  following  three  different  combinations  of  correlation  coefficients  for 
the  radar  suite  and  electronics  suite: 

■  Correlation  coefficients  of  0.5  among  the  radar  suite  components  and  0.2  between 
the  different  electronic  suite  components. 
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■  Correlation  coefficients  of  0.5  among  the  radar  suite  components  and  0.5  between 
the  different  electronic  suite  components. 

■  Correlation  coefficients  of  0.8  among  the  radar  suite  components  and  0.5  between 
the  different  electronic  suite  components. 

As  discussed  above,  positive  correlations  give  rise  to  broader  distributions,  which  reflect 
higher  uncertainty.  The  no  correlation  PDF  in  Figure  4  is  the  same  as  the  no  correlation  shown 
in  Figure  9.  They  do  not  appear  to  be  the  same  due  to  the  difference  in  scale  because  they  are 
from  separate  Monte  Carlo  simulations.  Although  the  Monte  Carlo  simulations  will  give  similar 
results  for  each  run,  they  will  not  be  identical. 
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Correlation  Effects 
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Figure  4.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 
Based  on  Different  Correlation  Effects  in  Cumulative  Probability  Form 

4.  MAIMS  Principle  Effects 

The  MAIMS  principle  is  modeled  in  this  paper  by  using  the  three-parameter  Weibull 
(20%,  50%,  80%)  distribution  function  and  predetermined  percentile  points  for  the  MAIMS  set 
points.  By  implementing  MAIMS  into  the  distribution  function,  any  value  less  than  the  money 
allocation  point  is  equal  to  the  money  allocation  value.  The  percentage  parameters  used  for 
MAIMS  are  the  50th  percentile  or  median,  the  mean,  and  the  80th  percentile  funding  levels.  A 
spike,  or  delta  function,  is  observed  in  the  MAIMS  modified  distributions  at  the  money  allocation 
points.  These  money  allocation  points  correspond  to  the  budget  allocated  to  the  WBS  cost 
elements  by  the  project  manager. 

The  MAIMS  modified  functions  are  modeled  by  using  the  following  equation: 

If  Distribution  Value  <  X,  then  X,  else  Distribution  Value. 

By  using  this  equation,  the  value  of  the  MAIMS  modified  distribution  will  never  be  less 
than  the  value  X. 
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D. 


Results 


1.  Effects  of  Distribution  Choice  on  Cost  Forecast 

The  first  distributions  modeled  were  the  single  electronics  suite  elements  with  different 
distributions.  For  the  purpose  of  this  paper,  the  element  ExComm  is  chosen  for  this 
explanation.  Figure  5  shows  the  cumulative  frequency  distribution  of  the  different  modeled 
distributions  for  the  ExComm  element.  The  triangular  and  lognormal  distributions  show  similar 
characteristics,  which  is  expected  since  the  lognormal  distribution  uses  parameters  taken  from 
the  triangular  distribution  (mean  and  standard  deviations).  Both  Weibull  distributions  show 
expected  characteristics.  The  Weibull  (20%,  50%,  80%)  definitely  indicates  a  more  pessimistic 
cost  forecast  because  as  the  cumulative  probability  increases,  the  cost  increases  more 
significantly  than  for  the  Weibull  (10%,  50%,  90%)  distribution.  This  overlay  indicates  that  the 
choice  of  distribution  used  in  modeling  plays  a  significant  part  in  results  obtained  for  cost.  The 
three-parameter  Weibull  distributions  represent  a  more  realistic  cost  outcome  for  high-risk 
components.  Weibull  distributions  allow  for  modeling  of  highly  complex  distributions  using  DFA, 
while  triangular  distributions  have  a  more  restrictive  shape,  making  it  difficult  to  fit  three  arbitrary 
percentiles  for  the  low,  most-likely,  and  high  values  (Kujawski  et  al.,  2004). 
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Figure  5.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Element 
ExComm  Cumulative  Frequency  Distributions  for  Different  Probability  Distributions 

The  probability  distribution  functions  shown  in  the  overlay  in  Figure  6  illustrate  expected 
behaviors  for  the  ExComm  PDFs.  Both  the  triangular  and  lognormal  distributions  are  narrow 
because  the  triangular  distribution  upper  and  lower  bounds  do  not  allow  for  infinite  upper  cost 
ranges.  The  sum  of  the  Weibull  (20%,  50%,  80%)  distributions  shows  a  more  pessimistic 
behavior  in  comparison  to  the  sum  of  the  Weibull  (10%,  50%,  90%)  distributions. 
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Figure  6.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Element 

ExComm  with  Different  PDFs 

Once  all  the  individual  electronics  suite  elements  are  modeled,  they  are  summed  up 
probabilistically  in  the  main  worksheet  in  Excel®  to  obtain  the  entire  electronics  suite  cost.  The 
simulation  selects  a  random  value  from  each  of  the  element  distributions  then  adds  them  to 
create  one  data  point  for  the  total  cost.  This  is  repeated  10,000  times  to  create  the  total  cost 
distribution.  When  all  the  distribution  functions  (assumption  cells  in  the  model)  of  the  electronics 
suite  elements  are  probabilistically  summed,  the  resulting  cost  is  illustrated  in  the  overlay  shown 
in  Figure  7.  All  four  distributions  have  the  appearance  of  a  normal  distribution  consistent  with 
the  Central  Limit  Theorem  (Garvey,  2000). 

The  lognormal  and  triangular  distribution  functions  give  rise  to  relatively  narrow  total  cost 
distributions,  consistent  with  the  finite  ranges  of  the  contributing  triangular  distributions  and  the 
modeling  of  the  lognormal  distributions  using  the  corresponding  mean  and  standard  deviation 
values.  The  Weibull  (10%,  50%,  90%)-based  cost  distribution  shows  more  narrow  behavior  for 
cost  range  than  the  Weibull  (20%,  50%,  80%)-based  distribution.  The  Weibull  (20%,  50%, 
80%)-based  distribution  allows  for  more  uncertainty  in  data  elicitation  from  SMEs.  Weibull 
distributions  not  only  show  higher  probabilities  of  cost  overruns  but  also  higher  probabilities  of 
cost  underruns.  These  underruns  reflect  the  assumption  of  10%  and  20%  as  the  low  value 
parameter  for  the  distribution,  rather  than  using  it  as  the  minimum  value.  Figure  8  shows  the 
same  data  as  Figure  7,  except  that  it  is  in  the  cumulative  probability  form. 
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Figure  7.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 

Based  on  Different  Distribution  Selections 
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Figure  8.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 
Based  on  Different  Distribution  Selections  in  Cumulative  Probability  Form 
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2. 


Effects  of  Correlation  on  Cost 


As  discussed  in  Section  II.B.1,  two  types  of  correlations  are  modeled  in  this  paper:  (1) 
Correlations  among  cost  elements  within  the  radar  suite,  and  (2)  correlations  among  cost 
elements  in  the  entire  electronics  suite.  This  paper  uses  the  following  three  correlation 
coefficient  factors  to  show  the  correlation  between  the  radar  suite  elements  and  the  rest  of  the 
electronics  suite  components: 

■  Radar  suite  correlation  coefficient  =  0.5,  electronics  suite  correlation  coefficient  =  0.5 

■  Radar  suite  correlation  coefficient  =  0.5,  electronics  suite  correlation  coefficient  =  0.2 

■  Radar  suite  correlation  coefficient  =  0.8,  electronics  suite  correlation  coefficient  =  0.5. 

The  choice  of  the  values  listed  above  simulates  an  environment  that  is  not  a  perfectly 
correlated  or  no-correlated  situation.  The  (0.5,  0.5)  correlation  assumes  there  is  an  equal 
correlation  relationship  between  the  subcomponents  of  the  radar  suite  and  the  elements  of  the 
electronics  suite.  The  (0.5,  0.2)  correlation  illustrates  the  effects  of  having  a  stronger  correlation 
between  the  elements  of  the  electronics  suite  than  between  elements  of  the  entire  electronics 
suite.  The  (0.8,  0.5)  correlation  shows  the  impact  of  a  stronger  correlation  between  components 
of  one  system  than  between  different  systems.  These  correlation  coefficients  represent  a 
limited  set  of  parameters  for  investigation  in  this  paper.  Further  research  in  the  determination  of 
appropriate  correlation  coefficients  and  their  effect  is  needed  to  provide  a  more  complete 
analysis. 

The  impact  of  correlation  effects  is  seen  in  Figures  9  and  10.  These  overlays  show  the 
different  distributions  that  are  a  result  of  a  1 0,000-run  Monte  Carlo  simulation  for  the  correlated 
distributions  in  cumulative  distribution  form  (Figure  9)  and  PDF  form  (Figure  10).  The  blue  PDF 
is  the  reference  Weibull  (20%,  50%,  80%)  distribution  with  no  correlation  effects.  This 
distribution  has  the  most  narrow  cost  range  when  compared  with  the  correlated  distributions. 

The  cost  ranges  of  the  Weibull  distribution  increases  as  the  correlation  factors  increase.  Also, 
in  the  cumulative  probability  distribution  shown  in  Figure  9,  all  the  distributions  intersect  at  the 
mean  value  for  the  cost. 
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Figure  9.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 
Showing  the  Impact  of  Different  Correlation  Effects 


CGX  Electronics  Cost 
Correlation  Effects 


Figure  10.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 
Based  on  Different  Correlation  Effects  in  Cumulative  Probability  Form 

As  expected,  the  (0.5,  0.2)  correlation  being  the  smallest  has  the  least  effect  on  the  total 
cost  distribution.  It  is  interesting  to  note  that  the  distribution  resulting  from  the  (0.5,  0.5) 
correlation  does  not  differ  much  from  the  (0.8,  0.5)  distribution,  but  both  of  these  correlations 
have  a  more  significant  effect  than  the  (0.5,  0.2)  correlation.  This  indicates  that  a  change  in  the 
correlation  factor  for  the  radar  suite  from  0.5  to  0.8  is  not  as  significant  as  a  change  in  the 
correlation  factor  from  0.2  to  0.5  for  the  different  components  of  the  entire  electronics  suite. 
These  results  suggest  that  the  correlation  effects  are  important  for  probability  values  midpoint 
between  the  mean  and  the  extremes,  but  there  is  little  difference  for  values  beyond  0.5.  The 
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results  in  Figure  10  are  consistent  with  theoretical  predictions  of  positive  correlation  effects  in 
that  the  total  cost  becomes  broader  than  for  uncorrelated  total  cost  (Kujawski  et  al. ,  2004). 
Further  investigations  are  recommended  to  quantify  correlation  effects. 


3.  MAIMS  Effects  on  Cost 

The  MAIMS  modified  cumulative  probability  and  density  density  distributions  for  the 
electronics  suite  cost  are  shown  in  Figures  11  and  12.  Characteristics  of  the  MAIMS  modified 
PDF  is  that  they  will  never  have  a  value  less  than  the  chosen  value  of  modification.  So,  for  the 
MAIMS  50th  percentile  modified  distribution  in  Figure  11,  the  distribution  has  no  value  less  than 
the  50th  percentile  baseline  cost  level.  In  Figure  12,  the  spikes  or  delta  functions  normally 
associated  with  the  individual  MAIMS  distributions  are  not  seen  as  they  are  modulated  when 
summed. 


CGX  Electronics  Cost 
Weibull  (20.  50.  80)  MAIMS  Effects 


Figure  11.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 
Showing  the  MAIMS  Effects  in  Cumulative  Probability  Form 
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Figure  12.  CG(X)  Crystal  Ball®  Analysis,  10,000  Runs,  Overlay  of  Electronics  Suite  Cost 

Showing  the  MAIMS  Effects  in  PDF  Form 

It  is  important  to  note  the  significant  rise  in  cost  curve  that  occurs  as  the  MAIMS 
modification  value  increases.  The  mean  value  of  the  distribtion  increases  as  the  funding  level 
increases,  and  this  is  very  clear  in  Figure  11.  The  curve  representing  the  MAIMS  80% 
distribution  shows  how  the  budget  is  always  high  when  comparing  it  to  the  MAIMS  50%  or 
MAIMS  mean  distribution.  This  effect  is  because  once  money  has  been  allocated  to  a  WBS 
element,  it  is  almost  never  seen  in  cost  savings  as  underruns  because  cost  account  managers 
never  return  money  to  the  project.  Any  remaining  money  from  one  WBS  is  subsequently  spent 
on  a  different  existing  WBS  that  has  cost  overruns. 

These  simulations  can  be  considered  with  other  cost  factors  in  making  program 
management  decisions  regarding  budgets.  Funding  projects  at  a  level  too  low  to  cover  costs 
will  lead  to  cost  overruns,  while  funding  at  a  level  that  is  too  high  leads  to  money  not  being 
recouped  as  savings  later.  Allocating  reasonable  budgets  is  the  goal. 

E.  Chapter  Summary 

The  research  for  this  paper  is  based  on  the  NAVSEA  05C  CG(X)  model  provided  by  Mr. 
Chris  Deegan  and  his  CG(X)  analysts.  The  CG(X)  model  encompasses  all  factors  considered 
for  cost  of  the  entire  program,  including  labor  rates,  material  cost,  overhead  cost,  planning  cost, 
and  other  factors.  Because  of  the  complexity  of  the  model  and  the  numerous  factors  to 
consider,  one  portion  of  the  model  was  chosen  for  analysis.  The  Electronic  Suite  and  its  nine 
elements  are  specifically  targeted  as  the  focus  for  analysis. 

The  steps  used  in  the  analysis  of  the  CG(X)  model  include: 

1 .  Analyze  the  cost  factors  used  by  NAVSEA  05C  to  develop  the  electronics  cost. 

2.  Analyze  the  PDFs  used  for  the  electronics  cost  elements. 

3.  Identify  what  data  elicitation  methods  were  employed. 

4.  Determine  if  correlation  factors  were  used  in  the  cost  analysis. 
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5.  Develop  cost  factors  to  be  modeled  in  a  new  model. 

6.  Decide  which  PDFs  to  use  in  the  new  model. 

7.  Develop  a  new  cost  model  using  correlation  factors,  chosen  PDFs,  and  MAIMS 
influenced  distributions. 

Identified  cost  factors  include  NAVSEA  05C’s  probability  distribution  choice — a  method 
used  for  developing  the  low,  most-likely,  and  high  cost  values  for  the  electronics  suite  elements, 
data  elicitation  methods,  and  correlation  effects.  This  paper  explores  the  methodology  in 
choosing  different  probability  distribution  functions  and  their  applicability  to  the  model. 
Specifically,  triangular,  lognormal,  and  two  variations  of  the  three-parameter  Weibull  distribution 
are  considered. 

Methods  of  data  elicitation  are  explored  and  the  use  of  a  DFA  method  is  recommended 
for  future  use,  although  the  research  in  this  paper  did  not  involve  data  acquisition.  To  simulate 
the  use  of  a  DFA  methodology,  Weibull  distributions  are  employed  to  account  for  uncertainty 
associated  with  SME  estimation  of  data.  A  Weibull  (10%,  50%,  90%)  distribution  is  used  to 
simulate  a  more  optimistic  view  of  the  uncertainty  of  data,  while  a  Weibull  (20%,  50%,  80%) 
distribution  models  a  more  pessimistic,  but  probably  more  realistic,  view  of  the  uncertainty 
associated  with  data  from  the  SMEs. 

Two  types  of  correlation  effects  are  considered  and  modeled  in  this  paper.  The  first  is 
the  correlation  between  subcomponents  of  the  radar  suite,  and  the  other  is  the  correlation 
between  the  elements  of  the  electronics  suite.  The  radar  suite  is  one  of  the  elements  that 
makes  up  the  electronics  suite.  Analysis  shows  that  a  more  significant  effect  is  experienced 
with  higher  correlation  between  the  elements  of  the  electronics  suite  than  between  the 
subcomponents  of  the  radar  suite. 

MAIMS  modified  probability  distributions  are  modeled  to  show  the  significance  of  budget 
allocation  level.  These  distributions  are  truncated  at  the  baseline  budget  with  a  delta  function  at 
the  baseline.  This  is  based  on  the  principle  that  once  a  budget  is  allocated,  money  is  almost 
never  seen  in  the  form  of  cost  under  runs  as  the  project  progresses.  As  the  MAIMS 
modification  value  increases,  overall  distribution  cost  rises  with  increasing  probability  of 
success. 

III.  Conclusions 

A.  Summary 

This  paper  begins  by  exploring  the  definitions  of  risk  and  how  it  applies  to  the  guidance 
set  forth  by  current  Navy  leadership.  Admiral  Gary  Roughead,  Chief  of  Naval  Operations, 
states  that,  “We  manage  risk”  (Roughead,  2007).  The  need  to  develop  effective  acquisition  and 
shipbuilding  methods  to  successfully  deliver  an  “affordable  future  fleet”  (McCoy,  2008)  is 
imperative  if  the  Navy  is  to  meet  the  goal  of  a  313-ship  Navy  by  2020.  Cost  risk  analysis  is  one 
tool  of  many  that  can  be  used  to  help  attain  this  goal. 

This  paper  then  proceeds  to  examine  the  probabilistic  cost  analysis  approach  that 
NAVSEA  05C  currently  uses  to  predict  new  naval  vessel  construction  costs  and  to  develop  a 
method  that  better  predicts  the  ultimate  cost  risk.  Cost  factors  analyzed  in  this  paper  include 
the  effect  of  data  elicitation,  distribution  choice,  the  impact  of  the  MAIMS  principle,  and  the 
effect  of  correlation  factors.  Data  elicitation  and  MAIMS  have  significant  impact.  Correlation 
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effects  vanish  at  the  minimum,  mean,  and  maximum  values.  PDF  selection  has  a  small  impact 
as  long  as  the  distributions  fit  the  three  specified  percentiles. 

The  model  provided  by  NAVSEA  05C  encompasses  all  aspects  of  the  ship’s  cost  and 
only  the  nine  elements  of  the  electronics  suite  were  chosen  for  analysis  in  this  paper.  Using 
data  obtained  from  SMEs  for  low-,  most-likely,  and  high-cost  values,  experiments  were 
conducted  for  the  noted  cost  factors  in  the  Excel®  Monte  Carlo  simulation  add-in  Crystal  Bair. 

Triangular,  lognormal,  Weibull  (10%,  50%,  90%)  and  Weibull  (20%,  50%,  80%) 
distributions  are  modeled  and  simulated  to  show  the  impact  that  each  distribution  can  have  on 
budget  considerations  for  program  managers.  Both  the  triangular  and  lognormal  distributions 
show  narrow  cost  ranges  when  compared  to  the  Weibull  distribution  cost  range.  The  Weibull 
(10%,  50%,  90%)  represents  a  more  optimistic  distribution  than  the  more  pessimistic  Weibull 
(20%,  50%,  80%)  distribution.  The  Weibull  (20%,  50%,  80%)  distribution  accounts  for  the 
optimism  bias  commonly  associated  with  SMEs.  Data  elicitation  effects  are  modeled  through 
the  use  of  the  Weibull  distributions. 

Correlation  among  cost  elements  in  the  electronics  suite  is  modeled  with  the  use  of  the 
Correlation  Matrix  function  in  Crystal  Ball®.  This  paper  uses  three  sets  of  two  correlation 
coefficients  to  model  the  correlation  between  the  radar  suite  elements  and  the  rest  of  the 
electronics  suite  components.  The  results  suggest  that  the  correlation  effects  are  important  for 
probability  values  midpoint  between  the  mean  and  the  extremes,  but  there  is  little  difference  for 
values  beyond  0.5.  Further  investigations  are  recommended  to  quantify  correlation  effects. 

MAIMS  principle  modified  distributions  are  modeled  with  the  50th  percentile  cost  value, 
mean,  and  80th  percentile  cost  value  to  show  the  impact  of  funding  at  these  different  levels. 

The  MAIMS  principle  is  based  on  the  observation  that  for  a  given  budget,  any  money  allocated 
is  considered  money  spent.  Very  rarely  are  cost  underruns  experienced  on  a  project  once  the 
budget  has  been  allocated.  The  MAIMS  modified  distributions  in  this  paper  show  the  impact  of 
either  under-funding  a  budget  or  over-funding.  Under-funding  leads  to  cost  overruns  and  over- 
funding  leads  to  an  overall  higher  cost,  since  money  allocated  is  unlikely  to  be  recouped. 

B.  Recommendations  and  Areas  for  Further  Research 

The  analysis  conducted  in  this  model  is  only  a  starting  point  for  improvements  in  the 
area  of  cost  analysis  for  naval  vessels.  Although  the  methodology  used  in  this  paper  provides  a 
framework  for  obtaining  more  accurate  predictions  of  cost  than  those  in  use  with  current 
probabilistic  cost  analysis,  more  work  is  required  to  develop  a  more  complete  and  tested  model. 
Recommendations  for  future  research  in  the  area  of  probabilistic  cost  analysis  for  shipbuilding 
include: 


■  Use  of  the  DFA  method  to  obtain  data  for  cost  assessment.  Recommend  eliciting 
data  from  SMEs  at  the  10th,  50th,  and  90th  percentiles,  at  a  minimum,  for  relatively 
optimistic  view  of  the  data  quality,  and  at  the  20th,  50th,  and  80th  percentiles  if  a 
more  pessimistic  view  of  the  quality  of  data  is  present.  Take  into  consideration  the 
overconfidence  of  estimates  provided  by  experts  in  their  field  and  use  this  knowledge 
when  calibrating  data  for  analysis. 

■  Select  flexible  and  realistic  probability  distribution  functions  for  cost  analysis.  Create 
probability  distribution  functions  from  historical  data  and  adjust  for  expected 
differences  in  new  programs. 
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■  Incorporate  the  use  of  correlation  among  cost  elements  of  a  system.  Aim  to  use  a 
range  of  correlation  coefficients  that  is  realistic.  A  reasonable  range  for  correlation 
coefficients  is  between  0.3-0. 6,  with  some  room  for  variation.  Overly  optimistic 
correlation  coefficients  that  assume  independence  and  overly  pessimistic  correlation 
coefficients  that  assume  perfect  correlation  rarely  exist  in  real  data. 

■  Use  the  “Money  Allocated  is  Money  Spent”  (MAIMS)  principle  to  model  budget 
management  behavior.  The  MAIMS  function  will  not  allow  the  system  cost  to  be  a 
lesser  amount  than  the  budgeted  cost  baseline. 

■  Investigate  further  capabilities  available  with  advanced  modeling  software  such  as 
Crystal  Ball®  or  @Risk. 

■  Incorporate  systems  engineering  methodologies  and  thinking  into  the  development  of 
probabilistic  cost  analysis.  Kujawski  et  al.  (2004)  state  that  this  is  the  single  greatest 
challenge  to  the  development  and  use  of  improved  cost  models. 

Continuing  with  the  development  of  improved  cost  models  is  an  important  step  in  helping 
the  Navy  to  ensure  the  successful  acquisition  of  the  313-ship  Navy  it  desires.  Improved  cost 
models  can  give  project  managers  the  ability  to  develop  more  realistic  and  successful  plans  for 
their  projects,  while  enabling  them  to  make  better  budget  decisions.  The  cost  analysis 
methodology  presented  in  this  paper  can  serve  as  a  starting  point  for  further  advanced  research 
in  this  area  that  can  be  used  by  different  programs  across  the  Navy. 
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Background 


•  CNO  Guidance  for  2007-2008 

-  “We  manage  risk.  We  will  identify,  analyze, 
mitigate  and  then  accept  risk,  appreciating 
that  we  must  always  consider  the  risks  in 
aggregate  across  the  entire  force.  Zero  risk  is 
not  achievable  nor  affordable.  We  must 
manage  risk  and  move  forward  to  accomplish 
the  mission  while  safeguarding  our  people 
and  infrastructure.” 


Background  (cont.) 


•  The  First  100  Days  from  Vice  Admiral 
McCoy 

-  Common  mission:  develop,  deliver,  &  maintain  ships  and 
systems  on  time  and  on  cost  for  the  Navy. 

-  NAVSEA  Strategic  Business  Plan:  Build  an  Affordable 
Future  Fleet,  “...we  must  take  definitive  steps  to  ensure 
future  ships  and  weapons  systems  are  affordable  to  build, 
maintain,  and  upgrade  over  their  service  lives.” 

-  To  Build  an  Affordable  Future  Fleet,  We  Will: 

-  Focus  on  reducing  acquisition  costs,  including  applying 
more  risk-based  decisions  to  specifications  and 
requirements... 


Research  Questions 


•  What  is  the  realism  of  the  cost  models  currently  in  use 
by  NAVSEA  05C  for  the  CG(X)  class  ship? 

•  How  does  data  elicitation  impact  the  cost  prediction? 

•  How  does  the  choice  of  distributions  in  the  model  affect 
the  predictions  of  the  cost  outcome? 

•  What  is  the  impact  of  the  MAIMS  principle  on  the  cost 
probability  distributions? 

•  How  can  the  current  CG(X)  model  be  improved  to 
provide  more  accurate  models  to  predict  cost  and  risk? 


CGX  Model  From  NAVSEA  05C 


MAMDJF  AoA  Lead  Ship  Production  Cost  Estimate  Risk  Analysis 

All  Costs  Reported  in  Year  of  award 

XI  ( 

Lead  Ship)  (TY11$M) 

WBS 

HRE 

XI  (Rec) 

%  Low 

Low 

Most  Likely 

High 

%  High 

Ordnance 

$ 

14.70 

$107.84 

$  86.30 

$ 

93.77 

$  165.74 

VLS 

S 

3  20 

$70.93 

95% 

$  58  60 

$ 

61.68 

$  101.57 

165% 

CIGS 

S  11.500 

$  31.97 

85% 

$  23  63" 

$ 

27  80 

$  55.60 

200% 

DLS 

S 

- 

$  4.93 

95% 

$  4  07 

$ 

429 

$  8  58" 

200% 

Electronics 

$  160.38 

$  804.53 

$  633.57 

$ 

748.19 

$  910.20 

Radar  Suite 

$ 

23.15 

$  257.49 

85% 

$  190.32 

$ 

223.90 

$  257.49 

115% 

X-band 

S 

- 

S  65.58 

95% 

S  62.30 

S 

65.58 

S  75.41 

115% 

S-band 

$ 

22.72 

S  154.61 

95% 

S  146.88 

S 

154.61 

S  177.80 

115% 

Cooling 

s 

043 

S  3.71 

95% 

S  3.53 

s 

3.71 

S  4.27 

115% 

Power  (included  in  Ship  Total) 

$ 

1.82 

S  55.89 

95% 

S  46.17 

s 

48.60 

S  55.89 

115% 

ExComm 

$ 

30.21 

$  154.44 

75% 

$  100  72 

$ 

134  29 

$  1 88  01 " 

140% 

TSCE 

s 

3.45 

$  86.61 

75% 

$  56.49 

$ 

75.32 

$  94  15 

125% 

IUSW 

s 

5.69 

$  30.38 

95% 

$  25  09 

$ 

2641 

$  27  73 

105% 

EW-IW 

s 

21.02 

$  56.10 

75% 

$  36.58" 

$ 

48  78 

$  60.97" 

125% 

EO-IR 

s 

- 

$  14.15 

95% 

$  1169 

$ 

12  31 

$  12.92 

105% 

IFF 

s 

- 

$  9  65 

95% 

$  7.98 

$ 

840 

$  8.82 

105% 

MS  EI&T  (SS  Only) 

$ 

60  89 

$  105.19 

80% 

$  73.18 

$ 

91.47 

$  109.77 

120% 

MS  EI&T  (CS  Only) 

3 

15  98 

$  90  52 

80% 

$  62  97 

$ 

78  71 

$  94.45  ^ 

120% 

My  Analysis 


The  Low,  Most  Likely,  and  High  values 
were  obtained  from  Subject  Matter  Experts 
from  Dahlgren. 

Most  Likely  value  corresponds  to  50% 
value  based  on  triangular  distribution. 

Conducted  individual  Monte  Carlo 
simulations  of  the  L/ML/H  values,  but  this 
ended  up  giving  normalized  values  very 
close  to  the  original  values. 


My  Analysis  (cont.) 


Original  L/ML/H  values  were  used  in  my 
model. 

Investigated  effect  of  choice  of  distribution 
on  cost. 

Investigated  impact  of  radar  component 
correlation  on  cost.  Expect  only  positive 
or  independent  correlation. 

Modeled  effects  of  MAIMS  principle  on 
cost. 


Effects  of  Distribution  Choice  on 

Cost 


Data  Elicitation  and  Distribtuion  Selection  Effects 
for  CGX  Electronics  Cost 


575  660  745  831  916  993 

Cost,  $M 


Effects  of  Distribution  Choice  on 
Cost  Based  on  Funding  Level 


Data  Elicitation  and  Distribution  Effects 
forCGX  Electronics  Cost 


584  667  751  834  918  993 

Cost,  $M 


Radar  Correlation  Effects  on  Cost 


CGX  Electronics  Cost 
Correlation  Effects 


> 

4— 


o 


485 


626 


767 


908 


Cost,  $M 


1049 


1176 


Radar  Correlation  Effects  on  Cost 


CGX  Electronics  Cost 
Correlation  Effects 


485  626  767  908  1049  1176 

Cost,  $M 


MAIMS  Effects  on  Cost 


CGX  Electronics  Cost 
Weibull  (20.  50.  80)  MAIMS  Effects 

120  - 


0.20 


1.00 


0.80 


0.60 


0.40 


0.00 

641 


698 


755 


812 


Cost,  $M 


868 


919 


MAI  MS  80% 
—  MAIMSMean 


MAIMS  Effects  on  Cost 


CGX  Electronics  Cost 
Weibull  (20.  50.  80)  MAIMS  Effects 


Cost,  $M 


Conclusions 


Using  Crystal  Ball  to  develop  models  to 
assess  cost  risk  can  be  a  useful  tool  in  risk 
management  of  ship  acquisition  and  building 
projects. 

Factors  such  as  data  elicitation,  distribution 
choice,  correlation  effects,  and  MAIMS 
principle  all  are  important  in  considering  risk 
&  cost  modeling. 

-  Data  elicitation  and  MAIMS  have  the  most 
significant  impact. 


Conclusions  (continued) 


Data  elicitation  factors  are  important. 
Analyze  how  data  is  obtained  from  SME  to 
account  for  errors.  This  paper  used 
Weibull  distributions  to  simulate  Direct 
Fractile  Assessment  method  of  elicitation. 


Conclusions  (continued) 


Weibull  distributions  are  more  realistic  for 
modeling  cost  than  lognormal  and 
triangular  distributions. 

-  Lognormal  &  triangular  have  narrower  cost 
ranges. 

Weibull  (20,  50,  80)  represents  a 
pessimistic  distribution  to  account  for  SME 
bias. 


Conclusions  (continued) 


Correlation  modeled  with  the  Correlation 

Matrix  in  Crystal  Ball  ®. 

-  More  important  between  elements  of  the 
electronic  systems  than  between  different 
components  of  only  one  element. 

-  Most  significant  for  probability  values  midpoint 
between  the  mean  and  the  extremes,  with 
little  value  beyond  0.5. 


Conclusions  (continued) 


MAIMS  is  modeled  with  50th  percentile 
cost  value,  mean,  and  80th  percentile  cost 
value. 

MAIMS  based  on  the  observation  that  for 
a  given  budget,  any  money  allocated  is 
considered  money  spent. 

Distributions  show  how  under-funding 
leads  to  cost  overruns,  while  over-funding 
leads  to  overall  higher  cost. 


Recommendations 


Use  the  DFA  method  to  obtain  data  for 
cost  assessment.  Elicit  data  from  SMEs  at 
10th,  50th,  and  90th  percentiles  for 
optimistic  systems  and  at  20th,  50th,  and 
80th  percentiles  for  more  pessimistic 
systems. 

Use  flexible  and  realistic  probability 
distribution  functions  for  analysis. 


Recommendations  (continued) 


Incorporate  the  use  of  correlation  among 
cost  elements  of  a  system  with  realistic 
coefficients.  Range  of  0.3  -  0.6  is 
reasonable  and  can  be  adjusted  for 
variation. 

Use  the  “Money  Allocated  is  Money  Spent 
(MAIMS)”  principle  to  model  budget 
management  behavior.  MAIMS  function 
will  not  allow  system  cost  to  be  a  lesser 
amount  than  the  budgeted  cost  baseline. 


Recommendations  (continued) 


Continue  to  utilize  the  functions  of 
modeling  software  such  as  Crystal  Ball  ® 
or  @Risk. 

Incorporate  systems  engineering 
methodologies  and  thinking  into  the 
development  of  probabilistic  cost  analysis. 


Questions? 


